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Estimating  Economic  Consequences  in 


Organizational  Effectiveness  Experiments 
1.  The  Problem 

A major  thrust  in  current  research  on  organizations  is  ex- 
perimentation with  new  forms  of  work  organization.  The  purpose  of  this 
research  is  to  increase  organizational  effectiveness  in  both  the  public 
and  private  sectors.  Two  dimensions  of  organizational  effectiveness 
which  have  received  most  of  the  attention  are  productivity  and 
psychological  outcomes  from  work.  This  paper  examines  methods  to 
evaluate  changes  in  productivity  from  an  organizational  intervention. 

Most  new  forms  of  work  organization  experiments  assume  that 
changes  can  be  brought  about  in  economic  dimensions  such  as  productivity. 
Increases  in  productivity  are  beneficial  to  management  and  lead  to 
greater  earning  opportunities  for  the  worker.  The  problem  posed  by  this 
assumption  is  how  to  determine  whether  productivity  has  increased. 

A cursory  review  of  the  current  literature  on  organizational 
effectiveness  will  show  that  there  is  no  substantial  evidence  to 
document  changes  in  productivity.  Clearly  there  are  a lot  of  claims. 

Both  consultants  and  managers  have  said  that  productivity  in  the  ex- 
perimental organizations  has  increased  (cf.  Glaser,  1975);  however,  the 
validity  of  this  information  is  questionable  given  the  propensity  of 
people  involved  in  a study  to  overestimate  the  results  (Gordon,  1975). 

"Substantial"  evidence  means  that  quality  data  is  available  over  a 
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sufficiently  long  period  of  time  and  that  there  are  sufficient  controls 
available  to  separate  out  the  effects  of  the  experimental  intervention 
from  other  variables  that  influence  productivity.  Most  studies 
(cf.  Glaser,  1975)  simply  assert  that  productivity  is  improved  or  in- 
creased by  some  percent  without  detailing  the  source  of  the  data  or 
identifying  the  cause  of  the  change.  The  basic  thesis  in  this  paper 
is  that  production  is  a function  of  a number  of  controllable  and  un- 
controllable variables.  Changes  in  production  can  be  affected  by  any 
combination  of  these  variables.  An  organizational  intervention  simply 
adds  another  variable  to  this  complex  production  function.  Separating 
its  effect  is  a complex  analytical  task. 

Why  have  we  not  made  more  progress  in  estimating  productivity 
differences?  One  reason  is  that  much  of  the  research  in  this  area  has 
been  done  by  social  scientists  who  are  more  interested  in  psychological 
than  economic  outcomes  of  work.  Another  reason  is  that  many  people  in- 
volved in  introducing  organizational  change  have  been  more  concerned 
with  the  process  of  change  than  with  the  outcomes.  In  other  cases 
companies  have  not  been  willing  to  make  economic  data  available,  or  the 
resources  have  not  been  available  to  analyze  economic  data.  Another 
reason,  and  the  central  rationale  lor  this  paper,  is  that  there  have  not 
been  good  analytic  models  available  to  assess  productivity  changes  in 
organizational  experiments.  That  is,  we  do  not  have  a good  methodology 
for  estimating  productivity  changes. 
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2.  The  Setting 

This  paper  examines  several  models  used  to  estimate  productivity 
differences  in  an  experimental  intervention  in  a coal  mine.  The  forms 
of  the  models  are  presented  at  a general  level  to  enable  the  models 
to  be  generalized  to  other  settings.  Clearly,  some  of  th  ■ variables 
in  the  models  will  change  in  different  settings,  but  the  form  of  each 
model  and  the  testing  procedures  will  not. 

The  Rushton  Mining  Company  entered  into  a Quality  of  Work  (QOW)  experi  nent 
in  1973  (Goodman  1977).  One  of  the  three  sections  (mining  departments)  beca ne 
an  experimental  section.  Autonomous  work  group  teams  were  created  in  (hat 
section  by  outside  consultants.  While  it  is  not  possible  to  detail  the 
experimental  intervention  (see  Goodman,  1976),  the  communication, 
decision  making,  authority,  and  reward  systems  of  that  section  were 
substantially  modified.  The  basic  hypothesis  was  that  these  changes 
would  increase  productivity  levels  and  the  quality  of  working  lift. 

The  design  of  this  study  permits  comparing  productivity  of  the 
experimental  section  against  two  olher  mining  sections.  tn  a sense 
these  sections  can  serve  as  control  groups  to  assist  in  isolating  the 
eftect  of  the  experimental  intervention  on  productivity.  However, 
since  these  three  sections  are  not  perfectly  matched  as  to  men, 
mac  tinery,  and  physical  conditions  the  design  is  not  truly  an  ex- 
perimental design.,  At  best  this  study  can  be  classified  as  a 
quasi-exper imental  design;  contrasts  within  the  experimental  section 
over  time  or  between  the  experimental  and  control  sections  cannot 
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definitively  isolate  the  experimental  effect  on  productivity. 

A coal  mine's  production  is  generally  stated  in  terms  of  tons 
produced  (per  section!.  The  set  of  variables  affecting  production  is 
complex.  Some  variables  are  controllable,  such  as  the  number  of  men 
working  or  the  type  of  equipment.  Other  variables  are  un< ontrollable 
such  as  the  character  of  roof  and  runway  conditions.  In  comparing 
uifferences  in  tons  produced  per  section  it  is  important  10  under- 
stand which  variables  contribute  to  production.  For  example,  if  the 
experimental  section  outproduces  the  control  sections  but  the  latter 
have  very  poor  roof  conditions,  then  the  difference  in  productivity 
may  be  caused  by  uncontrollable  ph/sical  conditions  or  the  experiment. 

To  analyze  productivity  differences  we  need  to  describe  a production 
function  which  includes  the  major  predictor  variables,  and  then  to 
identify  whether  there  are  shifts  in  the  coefficients  of  t hese 
variables  over  time. 

3.  Testing  Procedures 

Since  it  is  not  possible  to  control  nonexpeeimental  variables 
(e.g.,  roof  conditions)  which  dif 1 orentially  affect  the  experimental 
and  control  groups,  it  is  necessary  to  statistically  control  for  these 
differences.  We  have  adopted  three  alternative  methods  for  testing 
for  productivity  changes  after  correcting  for  ch  tnges  in  uncontrollable 
variables  across  the  experimental  and  control  sections.  (Where  not 
indicated  otherwise,  the  experimental  section  prior  to  the  experiment 
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and  the  nonexperitnent.il  sections  will  be  referred  to  .is  i he  control 
sec l ions ) . 

All  three  methods  seek  to  identify  differences  in  the  structure 
of  the  production  function  (i.e.,  the  relationship  betwei  n resources 
used  and  quantities  produced)  between  the  experimental  section  and  the 
control  sections.  Thus  we  began  by  estimating  a separate  production 
function  for  each  section  for  each  year.  The  appropriati  positive  or 
negative  signs  can  be  specified  a priori  for  almost  all  v iriables  in 
the  production  function  equation.  For  example,  output  should  be 
positively  related  to  working  time  and  to  the  roof  and  runway 
condition  variables.  Before  accepting  the  estimated  equation  for  an 
individual  section,  we  have  required  that  all  coefficients  which  are 
significantly  different  from  zero  have  the  predicted  sign.  In  the 
few  instances  in  which  initial  estimates  did  not  satisfy  the  require- 
ment, we  have  attempted  to  determine  whether  exceptional  circumstances 
(e.g.,  unusual  physical  conditions^  in  a particular  section  might 
have  given  rise  to  an  unreasonable  coefficient  estimate.  In  some 
cases,  dummy  variables  were  introduced  to  account  for  such  special 
circums tances. 

The  first  and  simplest  procedure  is  to  test  for  differences  in 
the  constant  term  in  the  production  function  across  sections. 
Differences,  if  any,  in  the  sections,  including  differences  caused 
by  the  experiment,  are  assumed  to  affect  only  the  constant  term. 

Given  this  assumption,  the  homogeneity  of  the  sections  in  the  control 
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gi oup  or  a subset  of  the  control  group  can  be  tested  by  an  analysis 
of  variance,  i.e.,  by  testing  whether  a slgnl 1 leant ly  higher  pro- 
portion of  variance  Is  explained  by  allowing  a different  constant- 
term  for  each  section  than  by  Imposing  the  same  constant  term  across 
all  sections.  If  the  experiment  enhanced  productivity,  this  would 
be  reflected  by  a higher  constant  term  in  the  production  function 
for  the  experimental  section  than  in  the  production  functions  for 
the  control  sections.  The  _t-test  can  be  applied  to  determine 
whether  the  difference  in  the  constant  term  between  the  experimental 
and  control  groups  is  significant . 

The  above  approach  has  the  advantage  of  being  both  simple  to 

apply  and  easy  to  interpret.  The  disadvantage  of  the  procedure  is 

the  assumption  that  the  coefficients  on  both  the  controllable  and 
uncontrollable  variables  are  the  same  across  the  experimental  and  control 

sections.  It  is  not  obvious,  a priori,  that  this  should  be  the  case.  For 

example,  if  the  result  of  the  experiment  is  that  the  men  work  more  efficiently 

in  the  presence  of  adverse  conditions,  this  would  be  reflected  on  the 

coefficients  of  the  conditions  variables  rather  than  the  intercept.  The 

two  additional  tests  described  below  do  not  require  such  restrictive 

assumptions . 

A second  method  is  to  test  for  differences  in  any  of  the  estimated 
i or  I I I c lent s ol  l hi-  production  I unction  across  the  sections.  The 
maintained  (null!  hypothesis  is  that  all  the  coefficients  In  the 
production  function  are  the  same  in  all  sections.  The  alternative 


hypothesis  is  that  one  or  more  of  the  coefficients  differ  across 
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sections.  Thus,  if  the  experiment  resulted  in  a significant  change  in 
the  production  function  in  the  experimental  section,  the  maintained 
hypothesis  should  be  rejected  when  the  experimental  section  is 
compared  to  the  control  sections. 

The  maintained  hypothesis  in  the  first  step  of  the  test  is  that 
the  coefficients  in  all  six  sections  are  the  same.  The  alternative 
is  that  the  coefficients  in  all  six  are  different.  Since  the  tests 
are  sequential,  the  form  of  each  i ubsequent  test  will  depend  on  the 
outcome  of  the  one  preceding.  One  simple  sequence  is  as  follows: 

If  the  null  hypothesis  of  the  first  test  is  rejected,  the  production 
functions  of  the  sections  are  not  all  alike.  The  second  step  would 
then  be  to  test  whether  the  produ  tion  function  is  the  same  for  all 
control  sections.  If  so,  the  con  rol  gimps  would  be  combined  to 
yield  a single  production  function,  and  this  function  would  then  be 
coi  pared  to  the  experimental  section.  Selective  testing  of  subsets 
of  the  coefficients  would  determine  more  precisely  which  of  the  co- 
ef  icients  differ  between  the  experimental  and  control  section  pro- 
duc  tion  functions. 

A great  many  other  outcomes  are  possible.  The  control  sections 
might  be  found  to  differ  among  tl  emselves.  The  sect  ions  might  be  the 


The  probability  of  obtain  ng  a given  conclusion  is  not  in- 
dependent of  the  test  sequence.  Ideally,  we  would  like  to  compute  the 
piobability  of  obtaining  a particular  conclusion  taking  account  of  the 
ti st  sequence.  However,  in  the  absence  of  knowledge  of  the  true  values 
ot  the  parameters  of  the  model,  such  a calculation  is  not  possible.  Thus, 
ccnfidence  regions  for  a given  level  of  significance  are  determined 
b\  treating  each  step  of  the  tesi  sequence  as  independent  of  the  outcomes 
o previous  steps  in  the  sequence  This  rather  unsatisfying  procedure 
appears  to  be  widely  employed  since  no  generally  applicable  alternative 
p.  ocedure  is  available. 
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sanie  in  each  year  and  different  across  years.  The  control  and  ex- 
perimental sections  might  be  found  to  be  all  alike.  Rather  than 
enumerate  all  possible  sequences,  we  will  defer  further  discussion 
until  the  results  section  where  the  actual  test  sequence  is  presented. 

The  second  testing  procedure  described  above  also  has  several 

limitations.  The  power  of  the  analysis  of  variance  test  is  much 

greater  when  one  uses  a single  production  function  for  the  five 
2 

control  sections  in  a test  against  the  experimental  section  than  when 
one  tests  the  five  control  sections,  each  with  a different  production 
function,  against  the  experimental  section.  For  a given  level  of 
significance,  one  may  get  ambiguous  results  from  the  analysis  of 
variance  test  because  the  power  of  the  test  changes  when  different 
combinations  of  coefficients  or  sections  are  tested.  A second 
problem  is  that  individual  coefficients  may  be  different  in  the  ex- 
perimental section  relative  to  the  control  sections,  but  some  may  be 
higher  and  others  lower  so  that  the  net  effect  on  productivity  would 
not  be  clear  cut. 

The  third  method  used  to  test  the  experimental  section  against 
the  control  sections  is  designed  lo  provide  an  alternative  which 
would  potentially  be  conclusive  if  the  analysis  of  variance  test  were 


Here  each  set  of  annual  data  is  counted  separately  since  a 
separate  production  function  is  estimated  for  each  section  for  each 
year. 
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ambiguous  for  either  of  the  reasons  identified  above.  This  pro- 
cedure requires  only  the  production  functions  for  the  control  sections. 
The  values  of  the  variables  observed  in  the  experimental  section  are 
substituted  into  the  estimated  production  functions  for  the  control 
sections.  This  provides  a prediction  of  the  amount  which  would  have 
been  produced  had  the  resources  from  the  experimental  section  been 
used  in  the  control  sections  under  similar  physical  conditions.  The 
actual  average  weekly  production  from  the  experimental  section  can 
than  be  compared  to  the  predicted  average  from  the  equations  for  the 
control  sections.  If  the  predicted  amount  from  the  control  sections 
was  significantly  1 )wt  (higher)  than  actual  production  from  the 
experimental  section,  one  would  conclude  that  under  similar  physical 
conditions  the  experimental  section  was  more  (less)  productive  than 
the  control  sections.  The  relevant  test  statistic  has  the  dis- 
tribution. The  derivation  of  the  test  statistic  is  provided  in  the 
Appendix. 

Relative  to  the  second  method,  the  third  method  of  comparison 
has  the  advantage  of  simplicity;  the  information  on  productivity  is 
summarized  in  a single  statistic  comparing  mean  actual  output  of  the 
experimental  section  with  mean  predicted  output  from  using  the  same 
resources  in  the  control  section.  The  second  advantage  relative  to 
both  of  the  alternative  tests  is  that  one  need  not  estimate  a pro- 


' 


duction  function  for  the  experimental  section.  For  purposes  of  this 
test,  any  shifting  of  the  production  function  of  the  experimental 
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section  will  be  reflected  in  the  production  of  that  section.  Finally, 
the  power  of  the  test  Is  greater  than  the  analysis  of  variance  test 
because  the  latter  test  compares  'lie  sections  along  several  dimensions 
(as  many  dimensions  as  there  are  estimated  coefficients)  while  the  former 
test  is  based  pn  a single  dimension  (production). 

The  evaluation  results  are  not  based  exclusively  on  any  one  of  the 
above  tests.  The  multidimensional  nature  of  the  analysis  of  variance 
test  enables  one  to  identify  differences  among  the  sections  which  may 
not  be  reflected  in  the  production  figures.  These  differences  are  of 
interest  in  themselves  as  a supplement  to  the  simpler  test  of  differences 
in  productivity.  The  test  for  differences  in  the  constant  terms  is  the 
logical  first  step  in  attempting  to  pinpoint  differences  in  the  pro- 
duction functions. 

4.  Methodology 

Sample 

The  analyzed  data  set  consists  of  92  weeks  in  1973  and  1974  for  the 
three  sections--experiment al  (South),  North,  and  East.  The  sample  year 
1973  starts  at  1/2/73  and  ends  at  12/1/73.  From  these  48  weeks  two 
were  deleted  which  represent  the  miners'  vacation  period  in  July. 
Therefore,  the  1973  data  are  based  on  46  weeks. 

The  experimental  year  begins  in  the  first  week  of  December  1973 
and  runs  through  11/9/74.  From  these  49  weeks  the  two-week  vacation 
plus  another  week  lost  from  the  memorial  week  strike  (8/17/74  to 
8/24/74)  were  deleted  leaving  46  weeks  for  analysis.  In  the  re- 
maining period  in  November  and  in  the  first  week  of  December  there 
was  no  production  because  of  the  national  coal  strike.  Since  there 
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wore  only  .1  low  remaining  actual  production  weeks  In  becember  /mil 
since  1 1 ic*  work  In  that  period  was  under  a new  coni  rut,  we  decided 
to  include  those  weeks  in  the  1975  data. 

Variables 

Table  1 describes  the  major  variables  used  in  the  analysis.  The 
label  or  acronym,  variable  name,  and  description  are  given;  then  the 
operational  form  and  source  of  the  data  are  identified.  The  means 
and  standard  deviations  are  for  all  three  sections  for  the  1973  .and 
1974  time  periods. 

Our  prior  expectations  about  the  coefficients  are  indicated  in 
Table  2.  Since  we  are  using  a linear  approximation  to  the  production 
function,  and  the  conditions  variables  may  shift  the  function  up  or 
down,  we  do  not  have  a priori  expectations  concerning  the  sign  of  the 
constant  term.  Improvements  in  conditions  should  increase  production 
as  indicated  by  the  positive  signs  on  the  physical  conditions 
variables  in  the  table.  The  moves  variable  is  designed  to  measure  the 
distance  of  the  miner  from  the  feeder  and  should  thus  be  negative. 
Since  pillaring  is  more  productive  than  developmental  mining  we  ex- 
pected a positive  effect  on  the  pillaring  variable.  Increases  in 
crew  size  should  enhance  production  giving  a positive  sign  to  the 
man-days  variable.  belays  were  expected  to  have  no  effect  beyond 
the  reduction  in  working  time.  Since  delays  have  been  deducted  from 


potential  crew  time  in  constructing  ACWOT,  we  expect  zero  coefficients 
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on  the  delay  variables  when  delays  are  entered  separately.  Crew 
time  should  have  a positive  effect  on  production. 

5.  Results 

The  production  function  estimates  are  presented  in  Table  3.  A 
comparison  of  the  coefficient  estimates  across  the  various  sections 
reveals  that,  where  coefficients  are  statistically  significant,  they 
do  have  the  anticipated  sign. 

We  now  turn  to  the  results  derived  from  our  tests.  The  first 
method  of  testing  will  serve  to  indicate  whether  there  were  differences 
in  any  of  the  intercepts  of  the  model  across  the  sections. 

In  row  1 of  Table  4 require  the  intercept  for  the  five  controL 

sections  to  be  the  same  and  test  lo  determine  whether  the  intercept  lor 
South  74  (experimental)  is  significantly  different.  The  estimated  difference 
of  -48.9  is  not  significant.  In  row  2,  all  of  the  section  intercepts 
are  allowed  to  be  different  with  South  73  used  as  a reference.  The 
coefficients  for  North  73  and  North  74  are  significantly  lower  than 
the  coefficients  for  South  73  at  the  five  percent  level,  and  the 
coefficient  for  East  74  is  significantly  lower  at  the  ten  perc  nt 
level.  This  result  indicates  that  the  intercepts  for  the  cont  ol 
group  are  different,  and  this  is  confirmed  by  row  3 of  Table  4 
which  includes  only  the  five  control  sections.  North  has  a 
significantly  lower  intercept  than  the  remaining  control  sections  in 
b 'th  years.  In  the  remaining  three  rows,  the  intercept  of  each 
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section  in  1974  is  compared  to  the  1973  intercept.  The  difference 
is  significant  only  in  North. 

The  results  in  Table  4 provide  no  evidence  that  the  intercept 
has  been  shifted  by  the  experiment.  This  conclusion  emerges  in  the 
first  row  when  South  74  is  compared  to  the  entire  control  group  and 
in  the  last  row  when  South  74  is  compared  to  South  73.  The  results 
in  Table  4 do,  however,  suggest  that  there  are  significant  differences 
among  the  intercepts  of  the  sections  in  the  control  group.  The 
results  of  method  two  below  provide  a more  general  test  for 
differences  in  any  of  the  coefficients  across  the  sections. 

In  row  1 of  Table  5 we  test  whether  individual  production 
functions  for  the  six  sections  fit  the  data  significantly  better  than 
a single  function  applied  to  all  : ix.  At  the  five  percent  significance 
level,  the  sections  are  not  signi  icantly  different.  Since  the  power 
of  the  F-test  varies  considerably  when  various  sections  are  combined, 
we  present  the  results  of  several  alternaiive  tests.  The  results  in 
the  second  row  indicate  that  the  t ontrol  sections  are  not  significantly 
different.  When  a single  functioi  for  the  control  section  is  tested 
against  the  experimental  section  in  row  3,  the  difference  proves  to 
be  significant  at  the  one  percent  level. 

Further  evidence  of  the  similarity  of  the  control  sections  is 
provided  in  rows  4 and  5.  In  row  4,  a single  function  for  the  three 
sections  in  1973  is  tested  against  separate  functions  for  each  section 
and  no  significant  difference  is  ound.  In  row  5,  the  combined  control 
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group  in  1973  is  compared  to  the  combined  control  group  in  1974 
and  again  no  difference  is  found. 

Row  6 indicates  no  significant  difference  across  experimental 
and  control  groups  in  1974.  However,  when  the  experimental  and 
control  groups  are  combined  in  1974  and  tested  against  the  combined 
control  group  in  1973,  a significant  difference  is  found  as  in- 
dicated in  row  7.  This  should  be  contrasted  to  the  results  in  5 
where  no  significant  differences  were  found  when  the  experimental 
section  was  not  included  in  1974.  Finally,  the  models  i or  each  of 
the  sections  are  compared  across  years  in  rows  8,  9,  and  10.  lere 
it  is  found  that  one  of  the  control  sections  differs  between  1973  and 
1974,  but  the  production  function  for  the  experimental  section  in  1974 
is  not  significantly  different  from  the  production  function  in  1973. 

Additional  tests  were  conducted  to  explore  possible  differences 
between  South  74  and  the  control  group.  In  these  tests  additional 
dummy  variables  were  included  to  allow  the  ROOI , RUNWAY,  and  PILLARING 
coefficients  to  differ  for  South  74  when  it  was  included  in  estimating 
a single  production  function  for  the  experimental  and  control  groups 
combined.  After  dilferences  are  allowed  in  tin  coefficients  of  the 
three  variables  identified  above,  the  production  functions  for  the 
experimental  and  control  sections  are  not  significantly  different. 

The  conclusion  of  the  F-test  ; is  that  thei e is  some  relatively 
weak  evidence  that  the  experiment  il  and  control  section?  are  different. 
The  differences  are  attributable  o the  ROOF,  RUNWAY,  aid  PILLARING 
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i'in'  II'  1 1- 1 fill  h . Those  dlllriencos  In  I la*  coefficient s ol  t ho  cond  1 1 I ohm 
v;ir  i.'iblcs  may  be  ;it  I r Ibut  able  t o lln>  experiment  though  such  dilleronces 
would  not  have  been  predicted  a priori.  For  example,  it  may  be  that 
the  experiment  caused  the  crew  to  work  at  the  same  productivity  level 
independent  of  runway  conditions  giving  rise  to  the  insignificant  co- 
efficient on  runway  in  Table  3 for  South  74.  The  higher  coefficient 
on  pillaring  indicated  greater  productivity  in  pillaring  in  the 
experimental  section.  If  the  higher  coefficient  on  roof  is  to  be 
attributed  to  the  experiment,  one  would  have  to  conclude  that  the 
experiment  made  productivity  more  sensitive  to  changes  in  roof  con- 
ditions. Since  the  coefficients  on  root  and  runway  in  South  74  differ 
in  opposite  directions  from  those  of  the  control  group,  sampling  errors 
rather  than  experimental  effects  may  be  the  cause  of  these  differences. 

The  results  of  the  test  of  intercepts  appear  in  some  respects  in- 
consistent with  the  results  of  the  analysis  of  variance  test.  The 
intercept  tests  indicated  no  significant  effect  of  the  experiment 
while  the  analysis  of  variance  tests  indicate  a significant  difference 
between  the  experimental  and  control  groups.  Also,  the  intercept  tests 
indicated  that  North  differed  significantly  from  the  other  control 
sections  while  the  analysis  of  variance  tests  suggest  that  what 
differences  exist  are  attributable  to  East. 

The  explanation  of  these  seemingly  inconsistent  results  is  trace- 
able to  the  underlying  assumptions  and  the  relative  power  of  the  tests. 


The  intercept  test  is  predicated  on  the  assumption  that  differences.  If 
any,  will  be  reflected  in  the  intercepts.  Offsetting  differences  in  tin 
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other  coefficients  may  not  be  picked  up  by  the  intercept  test.  In 
contrast,  the  analysis  of  variance  test  allows  for  quite  general 
differences,  but  the  discriminatory  power  of  the  test  is  lower  than 
that  of  the  intercept  test.  Clearly,  it  is  desirable  to  have  a single 
summary  measure  of  the  effects  of  the  experiment.  The  intercept  test 
is  unsatis factory  because  there  is  no  a priori  reason  to  expect  only 
the  intercept  to  be  affected.  The  analysis  of  variance  test  identifies 
differences  but  does  not  indicate  the  net  effect  of  those  differences 
on  productivity.  It  is  for  these  reasons  that  we  developed  the  third 
method  of  testing  for  differences  in  productivity. 

both  the  intercept  test  and  the  analysis  of  variance  test  cast 
doubt  on  the  hypothesis  that  all  si  cl  ions  in  the  control  group  have 
the  same  production  function.  Tlieielore,  we  conducted  the  means  test 
not  only  with  the  combined  control  group,  but  also  using  only  South  73 
as  the  control  group.  If  unobserved  variables  are  responsible  for 
differences  among  the  control  sections.  South  73  may  be  a more- 
reliable  control  since  such  unobserved  variables  may  differ  less  over 
time  for  a given  section  than  they  vary  across  sections. 

The  results  of  means  tests  ar>  presented  in  Table  6.  tor  the 
results  in  the  upper  half  of  the  I hie,  the  production  function  for 
South  73  was  used  to  obtain  predicted  average  weekly  production. 

This  prediction  is  obtained  by  sub:  tituting  the  independent  variables 
observed  in  each  week  in  South  74  nto  the  estimated  production 
function  for  South  73.  Ihe  averagt  of  these  predictions,  denoted  y , 
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was  then  subtracted  from  actual  average  production  in  South  74  to 
obtain  the  estimated  difference,  d.  The  difference  is  an  estimate  of 
the  amount  bv  which  average  weekly  production  in  South  74  exceeded 
the  amount  which  would  have  been  produced  had  the  same  resources  been 
used  under  similar  physical  conditions  In  South  73.  The  results  in 
the  bottom  half  of  the  Table  were  obtained  by  applying  the  same 
procedure  using  the  production  function  estimated  when  the  data  from 
all  the  control  sections  were  combined. 

The  results  indi:ate  that  average  tonnage  produced  per  week  in 
South  74  was  not  significantly  different  from  the  amount  which  would 
have  been  produced  had  the  same  resources  been  used  in  the  nonexper imental 
sections  under  similar  physical  conditions.  This  conclusion  is  obtained 
using  either  South  73  or  the  combined  nonexper imental  sections  as  the 
control . 


6.  Discussion-Summary 

The  general  results  do  not  indicate  any  increase  in  prouuctivity 
in  the  experimental  section  during  the  first  experimental  year.  The 
reasons  for  these  results  can  be  classified  as  methodologica 1 or 
conceptual.  One  methodological  problem  may  be  that  we  have  not 
stated  the  models  correctly.  That  is,  variables  might  be  missing  or 
some  of  the  selected  variables  incorrectly  operationalized.  In  re- 
viewing the  models  we  have  used  in  this  analysis  there  are  some  co- 


efficients that  appear  different  I rom  what  we  would  have  predicted 


-18- 

and  in  some  cases  we  have  introduced  dummy  variables  to  deal  with 
special  problems  (e.g.,  pe<  ial  phvsical  conditions)  not  captured  by 
our  regular  measures.  To  deal  with  this  issue  we  thought  carefully 
about  the  appropriate  conceptual  model.  We  did  not  let  either 
techniques  or  the  data  primarily  guide  our  analysis.  When  anomalies 
have  appeared  in  die  data  we  have  one  back  to  the  original  data 
sources  to  identify  the  actual  daily  events  that  were  going  on.  To 
avoid  ex  post  solution  from  this  more  detailed  analysis,  we  always 
tried  to  recast  these  findings  back  into  our  conceptual  framework. 

Another  methodological  issue  concerns  the  control  groups.  The 
lack  of  improvement  in  productivity  might  be  explained  by  the  fact  that 
the  other  sections  are  not  equivalent  to  the  experimental  section  and 
the  comparison  with  these  groups  i-.  inappropriate.  We  have  tried  to 
respond  to  this  question  by  developing  a model  that  will  be  genera lizablo 
across  sections.  Indeed,  the  anal  /sis  examining  the  production  functions 
across  the  different  sections  show;  a great  deal  of  similarity.  The 
other  analytic  appro.ch  has  been  lo  treat  each  section  as  its  own 
control  group.  That  is,  we  compared  the  performance  of  the  experimental 
section  in  the  baseline  with  its  performance  in  the  experimental  year. 

The  overall  strategy  then  is  to  ai  knowledge  that  there  is  a problem 
of  equivalence,  but  at  the  same  time  to  use.  analysis  procedures  and  1 i 
consider  alternative  control  groiq  s.  If  we  get  a consistent  picture 
across  these  different  strategies  we  can  feel  more  cert  tin  about  the 


validity  of  the  results. 
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A third  issue  concerns  problems  of  measurement.  Unreliability 
in  either  the  independent  or  dependent  variables  clearly  will  con- 
found the  analysis.  To  some  extent  this  problem  is  outside  of  our 
hands  because  we  do  not  have  the  resources  to  set  up  our  own  measurement 
system  for  the  economic  variables.  In  some  cases  we  have  analytically 
examined  some  of  the  ordinal  scales  used  for  measuring  physical 
conditions  to  assess  the  validity  of  t hose  measures.  for  the  dependent 
variable  we  intend  t o relate  the  current  figure  against  t'’her  measures 
of  the  same  phenomena  during  the  coming  year  to  get  a measure  of  con- 
current validity. 

Another  mea  ;urement  issue  concerns  potential  biases  in  the 
measures.  Here  we  would  find  a high  degree  of  consistency  ^reliability  1 
but  validity  would  be  low.  The  issue  is  that  certain  bosses  could 
have  overstated  prodm  tion  as  a reaction  to  the  experimental  induction. 
Bosses  in  the  experimental  section  might  have  wanted  to  make  the 
section  look  better  while  bosses  in  the  control  groups  might  have  over- 
stated production  to  make  their  section  look  comparable  to  the 
experimental  section.  Our  own  observations  of  this  problem  are  the 
following:  first,  if  overstating  production  figures  by  the  foremen 

occurred  it  was  generally  a reaction  to  higher  level  management's 
demands  for  greater  production.  Our  information  suggests  this  pressure 
was  in  evidence  during  the  baseline  and  experimental  years  and  is  a 
constant  across  sections.  Second,  our  observations  and  interviews 


with  the  foremen  in  the  experimental  section  indicated  that  overstating 


pro  due  I Ion  did  not  occur  an  .1  iimrt  ion  ol  the  experiment  ; il  might 
have  occurred  from  production  pressure.  Third,  there  was  some 
evidence  that  in  some  of  the  control  crews  production  may  have  been 
overstated  as  a reaction  to  the  experiment.  This  behavior  however  was 
not  widespread  and  should  not  have  affected  the  comparisons  when  we 
treated  t lie  experimental  section  as  its  own  control. 

The  fourth  issue  concerns  the  level  of  aggregation  used  in  this 
analysis.  Our  data  have  been  examined  on  a weekly  basis.  It  could 
be  argued  that  our  estimates  would  be  more  refined  il  we  moved  to  a 
daily  basis.  This  is  an  issue  we  hope  to  explore  with  the  data  from 
1975  and  I97t>. 

The  last  issue  concerns  the  time  period  for  analysis  and  this 
might  be  the  most  compelling  issue.  It  takes  a long  time  period  to 
identify  the  results  of  a major  change  effort  like  the  Rushton  project. 

We  are  only  evaluating  performance  during  the  first  year.  It  is 
clearly  appropriate  to  withhold  am  conclusions  about  productivity 
until  the  1975  and  197b  data  is  examined. 

7,  c om  lus  ions 

In  selecting  a procedure  for  evaluating  organizational  el tect iveness 
experiments,  the  following  criteri.  are  applicable.  First,  the  pro- 
cedure should  be  relevant,  that  is  it  should  offer  the  potential  of 
answering  questions  of  interest  concerning  the  effects  ot  the 
experiment.  Second,  the  condition  required  for  applicability  of  the 


procedure  shouLd  be  satisfied  by  the  experimental  environment. 


In  evaluating  the  economic  consequences  of  an  experiment,  the 
primary  question  of  interest  will  be  determining  whether  the  experiment 
increased  productivity.  If  a change  in  productivity  occurred,  one  may 
also  want  to  pinpoint  how  the  production  function  differs  after  the 
introduction  of  the  experiment,  hven  if  no  effect  on  productivity  is 
realized,  there  may  be  some  interest  in  testing  whether  the  experiment 
had  any  impact  at  all  on  the  production  process. 

Of  the  three  procedures  considered  in  this  paper,  t lie  two  most 
appropriate  for  testing  for  differences  in  productivity  are  the 
intercept  test  and  the  means  test.  While  both  tests  satisfy  the  first 
criterion,  the  means  test  will  generally  be  preferable  by  the  second 
criterion.  The  intercept  test  is  based  on  the  assumption  that  the 
experiment  affects  only  the  intercept  and  no  other  coefficients  in  t ie 
production  function.  there  was  no  i priori  reason  for  expecting  thi 
to  be  the  case  in  the  experiment  discussed  in  this  paper,  and  it  is 
doubtful  that  one  would  generally  hive  such  strong  prior  information 
about  the  way  in  which  an  experimen  might  poieniially  affect  the 
production  function.  The  neans  test  is  particularly  suitable  for 
experiments  in  which  the  elfects  on  the  production  function  are  un- 
certain since  the  test  requires  no  assumption  about  the  way  in  which 
the  production  function  is  changed. 

Whether  or  not  a change  in  productivity  occurs,  one  may  wish  to 


test  whether  the  production  function  is  changed  at  all  by  the 
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experiment.  In  addressing  this  question,  the  analysis  of  variance 
test  is  appropriate  since  it  is  intended  to  identify  differences  in 
any  or  all  coefficients  between  the  experimental  and  control  groups. 
Should  evidence  of  differences  be  found,  tests  for  differences  in 
subsets  of  the  coefficients  can  be  applied  to  pinpoint  the  differences. 


Table  1 


Variable  Descriptions 


ACRONYM 

Variable  Name 

Meaning  of  Variable 

Operaticnal  Form 

COALBO 

Coal  and  boney  height 
added  together 

Height  from  ceiling  to 
the  floor 

Inc  lies 

ROOF 

Roof  conditions 

Quality  of  roof 

1 to  5 scale 
1 very  bad  5 very  | 

RUNWAY 

Runway  conditions 

Qua 1 i ty  of  runway 

1 to  5 scale 
1 very  bad  5 very  | 

PILLARING 

Pil laring 

(See  Baseline  Report, 
1975) 

Number  of  shifts  pill* 
per  week 

ABSF.N 

Number  of  total  absences 

Number  of  total  absences 

Total  absences  per  we« 

MOVES 

Major  moves 

Measure  of  the  distance 
between  face  and  feeder 

Assumes  the  value  zer* 
the  weeks  with  a majol 
(i.e.,  a move  of  more 
299  minutes)  and  incri 
by  one  for  each  week  , 
without  a major  move  : 

AVMAN 

Average  man  days 

Average  man  days  worked 
per  day  of  the  week 

Total  man  days  worked! 
week /days  per  week  wol 

MOMINDEL 

Moves  and  miner  delays 

Moves  and  miner  delays 

j 

.loves  and  miner  delay! 
minutes 

COMDEL 

Combined  delays 

Combined  delays 

Car,  machinery,  boltel 
physical,  and  miscelll 
de lay s-minutes 

EXOUTDEL 

Autonomous  work  group 
obligation  and  outside 
delays 

Autonomous  work  group 
and  outside  delays 

Autonomous  work  group 
obligation  and  outsidl 
minutes 

ACWOT 

Actual  crew  working  time 

Actual  crew  working  time 

Maximum  possible  creM 
time  (=5850)-total  de! 
minutes 

DU 

Denotes  dummy  variable 

Introduced  for  periods 
of  abnormal  conditions 

One  during  weeks  whea 
abnormal  conditions 
prevail 

Table  1 


Variable  Descriptions 


Operational  Form 

Source 

Mean 

Standard 

Deviation 

Inches 

Company  Records 

64.0 

4.8 

1 to  5 scale 

1 very  bad  5 very  good 

Rating  from  superintendent 

3.9 

1.3 

1 to  5 scale 

1 very  bad  5 very  good 

Rating  from  superintendent 

3.5 

1.7 

Number  of  shifts  pillaring 
per  week 

Company  records 

1.13 

3.4 

Total  absences  per  week 

Company  records 

4.8 

4.0 

Assumes  the  value  zero  for 
the  weeks  with  a major  move 
(i.e.,  a move  of  more  than 
299  minutes)  and  increases 
by  one  for  each  week 
without  a major  move 

Created  by  analysts 
on  moves  delay 

based 

1.7 

1 .8 

Total  man  days  worked  per 
week/days  per  week  worked 

Created  by  analysts 
on  company  records 

based 

18.4 

2.6 

.loves  and  miner  delays- 
minutes 

Created  by  analysts 
on  company  records 

based 

452.9 

428.0 

Car,  machinery,  bolter 
physical,  and  miscellaneous 
delays-minutes 

Created  by  analysts 
on  company  records 

based 

782.2 

650.6 

Autonomous  work  group 
obligation  and  outside  delays- 
mi nutes 

Created  by  analysts 
on  company  records 

based 

79.3 

275.9 

Maximum  possible  crew  working 
time  (=5850)-total  delays 
minutes 

Created  by  analysts 
on  company  records 

based 

4231 .0 

894.0 

One  during  weeks  when 
abnormal  conditions 

Created  by  analysts 
on  company  records 

based 

— 

— 

prevail 


w 
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Variables  Included  in  Product  ton  Function 


Independent  V.ir  i nl>  I os 


Antii  i paled 
S i p.n 


CONSTANT 
CCVLBO 
RO  IF 
RL  IWAY 
ABSEN 
MOVES 
PII.LAK  INC. 
AVMAN 
M01 INDEL 
COM DEL 
ACWOT 


+ 

4 

4 


I 

0 

0 

4 


i 


Table  3 

Production  Model  II 
Medium  Aggregated  Delays 


East  73  North  73  South  73 


Coef fic ient 

T Value 

Coefficient 

T Value 

Coef fic ient 

T Value 

CONSTANT 

1845. 

0.8051 

CONSTANT 

-2800. 

-1.021 

CONSTANT 

-1392. 

-0.6773 

COALBO 

-49.33 

-1  . 514 

COALBO 

-1 1 .06 

-0.2450 

COALBO 

12.76 

0.4857 

ROOF 

-71.30 

-0.9155 

ROOF 

149.7 

1.194 

ROOF 

68.41 

0.7221 

RUNWAY 

178.7 

3.268 

RUNWAY 

458.5 

6.240 

RUNWAY 

-51  .53 

-0.4656 

ABSEN 

-12.47 

-0.5925 

ABSEN 

15.30 

0.3376 

PILLARING 

29.97 

0.4976 

MOVES 

-194.7 

-2.404 

MOVES 

40.05 

0.4176 

ABSEN 

-6.050 

-0.1500 

AVMAN 

45.21 

1 .472 

AVMAN 

108.9 

1 .274 

MOVES 

-20.28 

-0.3184 

EXOUTDEL 

-1294. 

-1.168 

DU  1 N73 

-1619. 

-2.103 

AVMAN 

46.97 

0 . 54  58 

MOMINDEL 

-0. 1949 

-0.7866 

EXOUTDEL 

-2.987 

-1.549 

DU  1-5  S73 

-2644. 

-4.719 

COMDEL 

-0.05737 

-0.3810 

MOMINDEL 

-0.1291 

-0.3559 

EXOUTDEL 

-2.357 

-0.9867 

ACWOT 

1.239 

11.35 

COMDEL 

-.0.08661 

-0.4794 

MOMINDEL 

-0 . 0604 

-0.2107 

ACWOT 

1.090  • 

6.984 

COMDEL 

0.0569 

0.2032 

ACWOT 

1 .011 

6.794 

RJ  =»  0.9125 
Dw  = 1.05 

East  74 
Coe  ' tc ient 

T Value 

R2  = 0.9098 
Dw  = 1 .917 

North  74 
Coefficient 

T Value 

R2  = 0.8088 
Dw  = 1.5 

South  74 
Coef  f ic ient 

T Value 

CONS  IANT 

-3238. 

-1.660 

CONSTANT 

-3896. 

-1.385 

CONSTANT 

-7947. 

-1.642 

COALBO 

-22.32 

-0.8761 

COALBO 

-14.73 

-0.5448 

COALBO 

44 . 62 

0.7431 

ROOF 

389.5 

2.987 

ROOF 

102.4 

0.9332 

ROOF 

441.1 

3.312 

RUNWAY 

1 1 .40 

0. 1 750 

RUNWAY 

127.6 

1 .484 

RUNWAY 

-’03.6 

-0. 9025 

PI  LIAR  INC 

22.11 

2.167 

ABSEN 

29.71 

1.208 

PILLARING 

110.  ’ 

4.388 

ABSEN 

30.73 

1 .888 

MOVES 

1 7.32 

0.2421 

ABSEN 

30.63 

0.6?° 

MOVES 

-32.47 

-1 . 198 

AVMAN 

148.4 

1.629 

MOVES 

-115.0 

-1.97 

AVMAN 

196.6 

3.008 

DU  23,39-43 

N74  -1168. 

-2.858 

AVMAN 

128.6 

1 .539 

EXOUTDEL 

-0. 3979 

-0.  .405 

EXOUTDEL 

-0.8868 

-0.4327 

EXOUTDEL 

-0.1092 

-0.4884 

MOMINDEL 

-0.2954 

-1.667 

MOMI  NT) EL 

0.1521 

0. 6654 

MOMINDEL 

0.01382 

0.0  507! 

COMDEL 

-0.0186 

-0.1123 

MBCDEL 

0.6526 

1.639 

COMDEL 

0.1283 

0.4777 

ACWOT 

R‘  = 0.8964 

1.000 

10.41 

COMDEL 

ACWOT 

R2  = 0.8621 

0.2175 

1.233 

0.9918 

7.330 

ACWOT 

R2  > 0.8322 

1.269 

6.492 

Dw  * 2.0 


Dv  » 1.59  7 


Pw  = 1.57 


Table 
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r 


Production  Model  II 
Medium  Aggregated  Delays 


North  73 


06 

.7 

.5 

30 

05 

.9 

19. 

987 
291 
08661 
090  ' 


South  73 


T Value 

Coefficient 

T Value 

Cdef f ic ient 

T Value 

-1.021 

CONSTANT 

-1392. 

-0.6773 

CONSTANT 

-2287. 

-3.448 

-0.2450 

CQALB0 

12.76 

0.4857 

COALBO 

-4.247 

-0.4983 

1.194 

ROOF 

68.41 

0.7221 

ROOF 

75.29 

2.244 

6.240 

RUNWAY 

-51.53 

-0.4656 

RUNWAY 

155.9 

5.952 

0.3376 

PILLARING 

29.97 

0.4976 

PILLARING 

45.27 

4.162 

0.4176 

ABSEN 

-6.050 

-0.1500 

ABSEN 

17.01 

1.783 

1 .274 

MOVES 

-20.28 

-0.3184 

MOVES 

-82.45 

-2.970 

-2.103 

AVMAN 

46.97 

0. 5458 

AVMAN 

83.15 

5.475 

-1.549 

DU  1-5  S73 

-2644. 

-4.719 

DU  1 N73 

-1638. 

-2.705 

-0.3559 

EX0UTDEL 

-2.357 

-0.9867 

DU  RUNW  5 N73 

1079. 

6.334 

-0.4794 

MOMINDEL 

-0.0604 

-0.2107 

DU  1-5  S73 

-1729. 

-6.016 

6.984 

COMDEL 

0.0569 

0.2032 

DU  23,39-43  N74 

-1341 

-5. 133 

ACWOT 

1 .011 

6.794 

COMDEL 

-0.0218 

-0.3006 

MOMINDEL 

-0.1663 

-1.684 

EXOUTDEL 

-0.3037 

-2.160 

MBCDEL 

0.1307 

0.4083 

ACWOT 

1.143 

20.39 

R2  = 0.8088 

R2  = 0.8537 

Dw  = 1.5 

South  74 

Dw  = 1 .400 

r Value 

Coef  f ic ient 

T Value 

■1.385 

CONSTANT 

-7947. 

-1 .642 

•0.5448 

COALBO 

44.62 

0.7431 

0.9332 

ROOF 

441.1 

3.312 

1 .484 

RUNWAY 

-103.6 

-0.9025 

1.208 

PILLARING 

110.  ’ 

4.388 

0.2421 

ABSEN 

30.63 

0.6?'’ 

1.629 

MOVES 

-115.0 

-1.97 

■2.858 

AVMAN 

128.6 

1.539 

■0.4327 

EXOUTDEL 

-0.1092 

-0.4884 

0.6654 

MOMINDEL 

0.01382 

0.05075 

1.639 

COMDEL 

0.1233 

0.4777 

0.9918 

ACWOT 

1.269 

6.492 

Dw  =■  1 .57 
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Tests  for  Differences  in  Intercepts  Across  Sections 

Section  Coefficients 


Sections  Included 


In  Regression 

E73 

N73 

S73 

E74 

N 74 

S74 

ENSEN,  S74 

B** 

B 

B 

B 

B 

-48.9 

(.30) 

E73 ,N73 ,.373 
E74,N74,:>74 

-60.6 

(.35) 

-597.5 

(3.4) 

B 

- 108. 1 
(1.8) 

- ?.1  .3 
'2.0) 

-190.6 

(.99) 

E73 , N73 , S73 
E74.N74 

-9.1 

(.05) 

-535.7 

(2.4) 

B 

-103.4 

(.65) 

-i  19.9 
( 1.98) 

E73  ,E74 

B 

-115.4 

(.94) 

N73,N74 

B 

372.2 

(1.91) 

S73.S74 

B 

-163.9 

(.44) 


t-ratics  are  shown  in  parentheses. 

**  Sections  denoted  with  a B in  a given  run  were  used  as  a base 
against  which  the  remaining  section  intercepts  were  c unpared 


Significant  at  the  1%  level. 
Significant  at  the  5”  level. 
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Table  6 


Tests  for  Differences  Between  Predicted  and  Actual  Means 


Control:  South  73 


4619 


d = n - y 
e p 


97 


17 


Control:  Combined  Nonexperimental  Sections 


4708 


d = y - y 
e *p 


.03 


L 


Appendix 


DERIVATION  OF  THE  TEST  FOR 


DIFFERENCES  BETWEEN  MEANS 


The  purpose  of  the  test  developed  below  is  to  compare  mean  weekly 
output  between  the  experimental  and  control  sections  while  controlling 
for  differences  in  the  values  of  all  independent  variables.  The  goal 
is  to  place  as  few  restrictions  as  possible  on  the  form  and  properties 
of  the  production  function  in  the  experimental  section.  Operationally 
the  test  involves  substituting  observed  values  of  the  independent 
variables  from  the  experimental  section  into  the  estimated  production 
function  for  the  control  section.  The  test  then  compares  mean  weekly 
production  predicted  from  the  function  for  the  control  sections  to 
mean  actual  production  from  the  experimental  section. 

We  observe  the  values  of  the  dependent  variable  generated  in  the 
experimental  section.  Call  these  yg,  emd  let  the  mean  of  these  be 


(1)  ye  = E yet  • 

t=l  — 

We  assume  that  yet  is  normally  distributed  with  unknown  mean  p^ 

2 

and  variance  . In  general,  p^  t will  be  functionally  dependent  on 
values  assumed  by  the  vector  of  exogenous  variables  (Xt>  in  the 
experimental  section  at  time  t,  but  it  is  not  necessary  to  specify  the 
form  of  that  function  for  the  test  described  below.  Mean  output  in  the 
experimental  section  is  then  normally  distributed 


y.  ~ N(  p , e_) 

e e — 


2 


where 


T 

■ £ 

t-1 


e,t 


The  vector  of  values  (y^)  of  the  dependent  variable  In  the  control 
group  is  observed  with  a different  matrix  of  exogenous  variables  Z.  Let 
Z have  dimension  N x P;  there  are  N weekly  observations  on  P variables 
in  the  control  sections.  Let  X have  dimension  T x P where  there  are  T 
weekly  observations  on  the  vector  of  P independent  variables,  and  the 
ordering  of  variables  is  the  same  as  in  Z.  The  assumed  model  for  the 
control  group  is  linear: 

(3)  yc  - Z p + e 

2 

where  e ~N(0,  oc  IN)  and  Z and  e are  independent. 

If  the  matrix  X were  to  occur  in  the  control  section,  the  model 
in  (3)  implies 

(4)  y = X p + u 

~P 

where 

u ~N(0,  ct^  IT)  . 

That  is,  y^is  the  vector  of  observations  on  the  dependent  variable  which 
would  have  occurred  if  the  set  of  independent  variables  observed  in  the 
experimental  section  had  also  been  observed  in  the  control  sections. 

The  test  procedure  will  be  to  predict  y^  and  to  compare  the  mean  of  the 
prediction  to  yg . the  mean  of  the  observed  values  in  the  experimental 
section. 


1 


3 


By  taking  the  mean  of  the  observations  in  (4)  it  follows  that 


y ■ £ y . * X P + u 


where  u ~ N(0,  ) and  X is  the  vector  of  sample  means  of  the 


variables  in  X (i.e.,  X - £ X ). 

t-1 


yp~H(V 


where  u,  = X'8  . 

P ~ K. 

Since  p is  unobservable,  ordinary  least  squares  estimators  are  obtained 


from  (3) . 


P - <z'  Z)'1  V jrc 


The  mean  of  the  predicted  values  from  (4)  is  then 


a _ A 

yp  - V 5 • 


Since  ordinary  least  squares  is  unbiased,  it  follows  that 


E (y  ) - X*  p - u . 
P ~ ~ P 


The  variance  of  the  predicted  mean  is 

± ± A 

Var  (y  ) - E[(y  - u ) ' (y  - u ) ] 
P P P P P 

— A — A — 

- E £ (X  * (3  - X'  p)'(X’  p - X'  p)l 


o2  X'  (Z’  Z)'1  X . 

C _ ~ _ 


4 


Then 


(9) 


where 


VN(%'  °C  K) 


K - X'  (Z'  Z)'1  X . 


The  objective  is  to  test  for  the  difference  In  means 


(10) 


% * “e  ’ 


and  the  proposed  test  statistic  Is 


(11) 


a £ 

d = y - y . 
P e 


The  test  statistic  Is  unbiased. 

E(d)  = E(yp  - yfi)  =■  (ip  - Ue  ■ d 

The  variance  of  the  test  statistic  Is 

Var  (d)  - Var  (y  ) + Var  (y^)  =>  K o2  + 

T" 

where  the  assumption  that  u and  c are  Independent  has  been  utilized. 
Thus 

(12) 


<J~N(p  - p,  ,Ko^+o2) 
p e c e 

T 


The  goal  is  to  make  a minimal  set  of  assumptions  about  the 

experimental  section.  To  avoid  estimating  the  ^ which  are  functions 

e » t 

Then  the  variance  can  be 


2 2 2 

of  X , we  will  assume  that  a = a - a 
~t  e c 


estimated  using  the  data  from  the  control  sections.  The  standard 
error  of  the  regression  in  (7)  is 


(13) 


S2-  E(y  -9  )2 

c c c 

t-1  ct  t 

N-P 


5 


From  the  normality  of  e it  follows  that 


(14) 


(N-P)S 


<P 


The  ratio  of  a unit  normal  and  the  square  root  of  an  Independent 
chi-square  divided  by  its  degrees  of  freedom  is  distributed  as  the  t. 
Thus 


a * K+y 


' (N  - P)S 


=/»- 


N-P  . 


This  reduces  to  the  following 
d - d 


05) 


sdAT 


~t 


N-P 


This  is  the  basis  of  the  test  results  reported  in  the  economic  analysis 
of  mean  weekly  production. 


References 


Glaser,  Edward  M. , Productivity  Gains  Through  Worklife  Improvement. 

New  York:  Harcourt  Brace  Jovanovich,  1975. 

Goodman,  Paul  S.,  Report:  Rushton  Quality  of  Work  Project, 

Submitted  to  Quality  of  Work  Program,  Institute  for  Social 
Research,  University  of  Michigan,  1976,  1977. 

Gordon,  Gerald,  and  Edward  V.  Morse,  "Evaluation  Research,"  Annual 
Review  of  Sociology,  1975,  339-361  . 


- 


UISTRIIUIIION  1. 1ST 
LIST  l 

Library,  Code  2029 

U.S.  Naval  Research  Laboratory 

Washington,  l).C.  20390 


MANDATORY 

Of! ice  of  Naval  Research 
(Code  432) 

800  N.  Quincy  Street 
Arlington,  Virginia  22217 

D Lrec  tor 
U.S.  Naval  Research  Laboratory 
Washington,  D.C.  20390 
ATTN : Technical  Information 

Center 

Defense  Documentation  Center 
Building  5 
Cameron  Station 


Science  & Technology  Division 
Library  of  Congress 
Washington,  D.C.  20540 

Psychologist 

ONR  Branch  Office 

495  Summer  Street 

Boston,  Massachusetts  02210 


LIST  2 


Psycho  Logist 
ONR  Branch  Office 
1030  E.  Creet  Street 
Pasadena.  California  91106 


Research  Psychologist 
ONR  Branch  Office 
536  S . C la rk  Street 
Chicago,  Illinois  60605 


PR INC f PAL  INVEST i GATORS 


LIST  3 


Dr.  Earl  A.  Alluisi 
Old  Dominion  University 
Res.  Foundation 
Norfolk,  Vir  ini  a 23308 

Dr.  Judith  Daly 

Dec  s Lons  & Designs,  Inc. 

Suite  100 

8400  West  park  Drive 
Mel.  an,  Vir;;  :nia  22' 0J 

Dr.  1 o~  \.  Bty'-o.i 
D ; ” tnv  >t  of  Psych  logy 
Howard  University 
W i .hinglon,  D.(  . 2000L 


Dr.  H.  Russell  Bernard 
Department  oi  Sociology  & 
Anthropology 
West  Virginia  University 
Morgantown,  West  Virginia  26506 

Dr.  Arthur  Blaiwes 

Naval  Training  Equipment  Cent ir 

Orlando,  Florida  32S13 

Dr.  Milton  R.  Blood 
Schoo  l.  o C Hti  s i ness 
cco.’ia  Institute  of  Technology 
Atlanta,  Ccrgi.i  30332 


I 


Dr.  David  G.  Bowers 
Institute  for  Social  Research 
University  of  Michigan 
Ann  Arbor,  Michigan  48106 

Dr  . .lolm  .1  . Col  1 I ns 

Vice  1*  resident 

Essex  Corporal ion 

6305  Camtnito  Estrellado 

San  Diego,  California  92120 

Dr.  Harry  R.  Day 
University  City  Science  Center 
Center  for  Social  Development 
3624  Science  Center 
Philadelphia,  Pennsylvania  19104 

Dr.  C.  Brooklyn  Derr 
Associate  Professor,  Code  55 

MitXil  Post  Gr.a.'.u  te  School 

Monterey,  California  93940 

Dr.  George  T.  Duncan 
Carnegie-Me lion  University 
5000  Forbes  Avenue 
Pittsburgh,  Pennsylvania  15213 

Dr.  Fred  E.  Fiedler 
Department  of  Psychology 
University  of  Washington 
Seattle,  Washington  98105 

Dr.  Samuel  L.  Gaertner 
Department  of  Psychology 
University  of  Delaware 
2200  Wolf  Hall 
Newark,  Delaware  19711 

Dr.  William  E.  Caymon 
Suite  200 

1055  Thomas  Jefferson  St.,  NW 
Washington,  D.C.  2000/ 

Dr.  Paul  S.  Goodman 

Graduate  School  of  Industrial  Admin. 
Carneg  ic  -Me  1 Ion  i'n  Ivo  rs  i ty 
Pittsburgh,  Pennsylvania  15213 

Dr.  Gloria  L.  Grace 
System  Development  Corp. 

2500  Colorado  Avenue 

Santa  Moult  i,  California  90x06 


Dr.  J.  Richard  Hackman 
Administrative  Sciences 
Yale  University 
56  llillhouse  Avenue 
Now  Haven,  Connecticut  06520 

Dr.  Leo  A.  Hazlewood 
GAG  1. , Inc  . 

1815  Fort  Myer  Dr. 

Arlington,  Virginia  22209 

Dr.  Edwin  Hollander 
Department  of  Psychology 
State  University  of  New  York 
at  Buffalo 
4230  Ridge  Lea  Rd. 

Buffalo,  New  York  14226 

Dr.  Daniel  F.  Htick 

Cnn-arti  !'..jsear~':  Corp. 

WesLgate  Research  Park 
McLean,  Virginia  22101 

Dr.  Charles  L.  Hulin 
Department  ol  Psychology 
University  of  Illinois 
Champaign,  Illinois  61820 

Dr.  Rudi  Klauss 
Syracuse  University 
Public  Administration  Dept. 
Maxwell  School 
Syracuse,  New  York  13210 

Dr.  Edward  E.  Lawler 
Battelle  Human  Affairs  Research 
Center 

4000  N.F. . 41st  Street 

P.0.  Box  5395 

Seattle,  Washington  9$105 

Dr.  Aric  Y.  Lew  in 
Duke  University 
Duke  Stat  ion 

Durham,  North  Carolina  27706 

i>r  . Morgan  W.  McCall,  Jr. 
tenter  for  Creative  Leadership 
5000  Latwinda  Dr. 

P.0.  Box  P-l 

Greensboro , North  Carolina  27402 


L 


n 


Dr.  Terence  R.  Mitchell 
School  of  Business  Administration 
University  of  Washington 
Seattle,  Washington  98195 

Dr.  William  H.  Mobley 
CoJ lego  of  Business  Administration 
University  of  South  Carolina 
Columbia,  South  Carolina  29208 

Dr.  Thomas  D.  Morris 
The  Brookings  Institution 
1775  Massachusetts  Avenue,  NW 
Washington,  D.C.  20036 

Dr.  James  F.  Murphy 
National  Analysts 
A Division  of  Booz-Allen  & 

H am i 1 ton , Inc . 

*+00  Market  Street 
Philadelphia,  Pennsylvania  19106 

Dr.  Peter  G.  Nordlie 

Human  Sciences  Research,  Inc. 

7710  Old  Springhouse  Rd. 

McLean,  Virginia  22101 

Dr.  Herbert  R.  Northrup 
Industrial  Research  Unit 
University  of  Pennsylvania 
Philadelphia,  Pennsylvania  19174 

Dr.  A.  F.  K.  Organski 
306S  Institute  for  Social  Research 
University  of  Michigan 
Ann  Arbor,  Michigan  43104 

Dr.  Paul  Pedersen 

Society  for  Intercu 1 tural  Education, 
Training  and  Research 
107  MIB,  University  of  Pittsburgh 
Pitt  '.burgh,  Pennsylvania  15260 

Dr.  Chester  M.  Pierce 
Harvard  University 
N i cho l .louse 
Appiau  Way 

Cambridge,  Massachusetts  02138 

Dr.  Manuel  Ramirez 
Systems  and  Evaluations 
232  Swanton  Blvd. 

Santa  Cruz,  California  95060 


Dr.  Irwin  Sarason 
Department  of  Psychology 
University  of  Washington 
Seattle,  Washington  98195 

Dr.  John  Ruhe 

University  of  North  Carolina 
Dept,  of  Business  Admin. 

Charlotte,  North  Carolina  28223 

Dr.  Irwin  Sarason 
Department  of  Psychology 
University  of  Washington 
Seattle,  Washington  98195 

Dr.  S.  B.  Sells 

Texas  Christian  University 

Fort  Worth,  Texas  76129 

Ur.  li.  kaLic.e  ktaika 

Program  Director 

Manpower  Research  & Advisory  Services 
Smithsonian  Institution 
801  N.  Pitt  St.  - Suite  120 
Alexandria,  Virginia  22314 

Mrs.  Alice  I.  Snyder 

Mental  Health  Clinic 

Naval  Regional  Medical  Center 

Pearl  Harbor 

FPO  San  Francisco  96610 

Dr.  Paul  Spector 

American  Institutes  for  Research 
Foxhall  Square 
3301  New  Mexico  Avenue  N.W. 
Washington,  D.C.  20016 

Dr.  Richard  Steers 

Graduate  School  of  Management  & 

Bus  iness 

University  of  Oregon 
Eugene,  Oregon  97403 

Dr.  Lo  rat  id  B.  Szalay 

American  In.  ! • tv  cs  for  Research 

' uxh a II  Si,  i.tc 

3301  Net/  Mexico  Avenue  N.W. 

Washington,  D.C.  20016 

Dr.  V i c to r H . V room 

School  oi  Organizational  Management 

Y.a  I c University 

56  Hill  house  Avenue 

Nee'  Hawn  , Coe:  ct  i 'tit  06520 


3 


Dr.  Taul  Wall 

Division  of  Behavioral  Science 
Tuskegee  Institute 
Tuskegee , Alabe  'a  36088 

Dr.  Abraham  R.  Wagner 
Analytical  Assessments  Corp. 

357  South  Robertson  Blvd. 
Beverly  Hills,  California  90211 


Dr.  J.  Wilkenfeld 
Department  of  Government  & 

Politics 

College  Park,  Maryland  20742 

Dr.  Davis  B.  Bob row 

University  of  Maryland 

Department  of  Government  and  Politics 

College  Park,  Maryland  20742 


MISCELLANEOUS 


LIST  4 


Military  Assistant  for 
Human  Resources 
OAD  (E  & LS)  ODDR&E 
Pentagon  3D129 
Washington,  D (’ . 20301 

Air  University  Library/ 
LSE-81 10 

Maxwell  AFB,  A1  >bama  36112 

Dr.  A.  L.  Slafkosky 
Scientific  Advisor 
Commandant  of  t lie  Marine 
Corps  (Code  RD-1) 
Washington,  D.C.  20380 

LCd r . C.  A.  Pat  mi,  U.S.N. 
Director,  Humae  Goals  Dept. 
Cotie  70,  Naval  'training  Ctr. 
Orlando,  Florid.1  32813 

Navy  Personnel  R & 1)  Center 
Code  10 

San  Diego,  California  92152 

Human  Resources  Management 
Center  Attachment. 

Naval  Support  tivity 
c/o  FPO  New  York,  NY  09521 
ATTN:  TDC  Nels 

Jouvn'l  Supple  t Abstract 
Serv ice 

A PA 

1200  - l 7th  S'  ■ at  N.W. 
Washington,  D . 20036 


HQ,  USAF 
AFMPC/DPMYAR 

Randolph  AFB,  Texas  78148 

Army  Research  Institute 
C-.  ■i-.xionwea  in  8 1 -ig . 

1300  Wilson  Blvd. 

Rosslyn,  Virginia  22209 

Chief  of  Naval  Personnel 
Assistant  for  Research 
Liason  (Pers-Or) 

Washington,  D.C.  20370 

Training  Officer 

Human  Resource  Management  Ctr. 

NTC , San  Diego,  California  92133 

Office  in  Charge  (Code  L5 ) 

Naval  Aerospace  Medical 
Research  Lab. 

Naval  Aerospace  Medical  Center 
Pensacola,  Florida  32512 

HumRRO  (ATTN:  library) 

300  N.  Washington  Street 
Alexandria,  Virginia  223L4 

Division  Director  for  Social 
Sc icnce 

National  Science  Foundation 
1)00  G Street  N.W. 

Washington,  D.C.  00530 

AFOSR  (MI.) 

1400  Wilson  Blvd. 

Arlington,  Virginia  22209 


4 


Chief,  Psychological 
Research  Branch 
U.S.  Coast  Guard  (C-P-l/62) 

400  - 7th  Street  S.W. 
Washington,  D.C.  20590 

Bureau  of  Naval  Personnel 
(Pers-6) 

Assistant  Chief  of  Naval 

Personnel  for  Human  Goals 
Washington,  D.C.  20370 

Scientific  Director 

Naval  Health  Research  Center 

San  Diego,  California  92152 

Capt.  Bruce  G.  Stone,  USN 
(Code  N-33) 

Director,  Educ . & Training 

Research  c Prog.  Development 

CNET  Staff 

Naval  Air  Station 

Pensacola,  Florida  32508 


Director  of  Research 
HumRRO  Division  #4  (Infantry) 

P.0.  Box  2086 

Fort  Benning,  Georgia  31905 

Hr.  Luigi  Petrullo 
2431  N.  Edgewood  Street 
Arlington,  Virginia  22207 

Navy  Materiel  Command 
Employee  Development  Office 
Code  SA-65 

Room  150  Jefferson  Plaza 

Bldg.  #2,  1429  Jeff  Davis  Highway 
Arlington,  Virginia  20360 

Personnel  Research  and  Development 
Center 

United  States  Civil  Service 

Comm  is  s ion 

Bureau  of  Policies  and  Standards 
Washington,  D.C.  20415 

Department  of  the  Air  Force 

Air  Force  Institute  of  Technology  (AU) 

AFIT/SLGR  (LT  Col  Urns  tot) 

Wright-Pat terson  Air  Force  Base, 

Ohio  45433 


